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Abstract 

This paper presents a statistical analysis to support the notion that the Information Systems Analyst 
(ISA) exam can be used as a program assessment tool in addition to measuring student performance. 
It compares ISA exam scores earned by students in one particular Computer Information Systems 
program with scores earned by the same students on the Major Field Test (MFT). The paper shows 
that the ISA Exam appears to measure knowledge retention in the same manner as the MFT. Since 
the MFT is recommended as a valid program assessment tool, the ISA Exam should be similarly 
recommended. 
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1. INTRODUCTION 

The Information Systems Analyst (ISA) Exam is 
designed as an exit examination over the 
material covered by the IS2002 Model 
Curriculum that is jointly sponsored by the 
Association for Computing Machinery (ACM), 
Association for Information Technology 
Professionals (AITP), and Association for 
Information Systems (AIS). It is jointly 
administered by the Institute for Certification of 


Computer Professionals (ICCP) and one of ICCP's 
divisions, the Center for Computing Education 
Research (CCER). Those who hold a bachelor 
degree and score sufficiently highly on the ISA 
exam qualify to be certified as an ISA- 
Practitioner (50% score) or ISA-Mastery (70% 
score) (CCER, 2010; ICCP, 2010). 

The ISA Exam is considered to be a good tool by 
which to assess students' learning in under¬ 
graduate CIS programs. Similarly, Educational 
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Testing Service's Major Field Test (MFT) (ETS, 
2010) is considered a good measurement of 
students' knowledge retention in specific 
academic disciplines. Moreover, the MFT is also 
considered to be a good measure of program 
assessment. 

The issue addressed by this article is whether it 
is valid to use the ISA exam for program 
assessment in addition to student assessment. 
Two references were found in the literature on 
this topic (Carpenter, et al., 2009; Segall, et al., 
2009). This article builds on the former by 
providing additional statistical support for the 
ISA exam as a program assessment tool. 

2. LITERATURE REVIEW 

Program assessment is a critically important 
activity within quality educational institutions. It 
measures the contribution that a program 
makes to students' learning, thereby insuring 
quality of graduates. Ultimately, though, 
Palomba & Banta (1999) insist the emphasis of 
program assessment is on programs rather than 
individual students. The Association to Advance 
Collegiate Schools of Business (AACSB) requires 
program assessment of its accredited programs 
in order to show evidence of program quality 
and to establish internal systems for quality 
improvement (AACSB, 2006). 

Program assessment is done in a variety of 
manners using a wide range of methodologies. 
Those include: interviewing stakeholders in the 
capstone course (Payne, Whitfield & Flynn, 
2002), recording multiple incidents or 
"occasions" throughout the program (Moberg & 
Walton, 2003), centering on course syllabi 
(Cunningham & Omaoayole, 1998), examining 
critical incidents throughout a program (Bycio & 
Allen, 2004), evaluating by student peers 
(Aurand & Wakefield, 2006), gathering feedback 
from alumni and employers (Dyke & Williams, 
1996), and focusing on competencies (Roberson, 
et al., 2002). Strong support for multifaceted 
methods of program assessment exists in Young 
(1996), Palomba & Palomba, (1999), 
Mirchandani, Lynch & Flamilton (2001). 

The Mirchandani, et al., multifaceted approach 
(2001) includes the use of the MFT exam. 
Others who support the use of the MFT as part 
of program assessment include Black & Duhon 
(2003), Karanthonos (1991), and Manton & 
English (2002). Black & Duhon (2003) go on to 
suggest the use of other objective tests that 
measure specific competencies. 


9(4) 

September 2011 


The ISA Exam is one such objective test that 
purports to measure the specific competencies 
included in the IS2002 Model Curriculum (CCER, 
2010). Many programs that adhere to that 
model curriculum do indeed use the ISA Exam 
as a measure of student performance in the 
program. That is endorsed by Kamoun & Selim 
(2008), Laudry, et al. (2008), and Wagner, et al. 
(2008). 

Only two references were found that advocate 
the use of the ISA Exam as means of program 
assessment in addition to assessment of 
individual students. Segall, et al., (2009) 
explain a database approach for matching the 
results of the ISA exam to the learning units of 
the IS2002 Model Curriculum. In that sense, 
the ISA exam results are indeed being used to 
assess the program itself as well as students 
within the program. 

Carpenter, et al. (2009) described a pre-test, 
post-test study using the ISA Exam for program 
assessment. In that 2003-2007 study, thirty- 
five students took the ISA exam twice, first as 
underclass members and second as graduating 
seniors. The mean ISA Exam scores for the 
second test were indeed statistically significantly 
higher than for the second test. That alone 
indicates that the intervening courses in that 
particular CIS program did add to the knowledge 
retention of those students. That is a positive 
indicator for the assessment of the program. 

In the Carpenter, et al. (2009) study, fifty-one 
students took both the ISA exam and the MFT. 
The means percentile scores for those students 
were seen to be equal. Their conclusion is that 
"the ISA and MFT have equal value in terms of 
their usability as valid measurements for 
assessment purposes." (p. 358). 

3. RESEARCH METHODOLOGY 

The logic that flows from the Literature Review is 
as follows. Since MFT is widely recognized as a 
valid means of program assessment, if the ISA 
Exam does indeed demonstrate student 
knowledge retention in the same manner as 
does the MFT, then the ISA Exam should be 
considered in similar light as a valid program 
assessment tool. 

The null hypothesis is that students who took 
just the ISA Exam performed statistically equally 
(using percentile scores) as those students who 
took both the ISA Exam and the MFT. Failure to 
reject the null hypothesis would indicate that the 
ISA measures student retention of knowledge 
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from the CIS courses in the same manner as 
does the MFT for business courses. 
Thus, hypothesis 1 (Hi) is: 

Hq'- M ISA &. MFT = FlSA 

Ha'- Fisa&mft ^ Fisa 

This study includes one hundred and three 
students in a computer information systems 
program in a western state college who took the 
ISA Exam between 2003 and 2010. Thirty-one 
of those students took only the ISA Exam. 
Seventy-two of the students took both the ISA 
Exam and the MFT. 

Next, the researchers must measure whether 
the ISA percentile scores of the two groups of 
students can be considered statistically equal. 
This would indicate whether there is some 
anomaly in one group or the other. Only then 
can the ISA and MFT percentile scores of those 
who took both exams be compared to each 
other. Thus, hypothesis 2 (H 2 ) is: 

Ho'- l 1 ISA — I 1 MFT 

Ha'- I 1 isa ^ Fmft 

Other interesting aspects were also explored. 
Specifically tests were run to determine what 
portion of the variability in the ISA Exam scores 
are explained by the MFT scores, and whether 
the students' graduating grade point average 
(GPA) explains any of the variability in ISA Exam 
scores was also considered. 

Data for the study were taken from reports 
provided to the college by the Educational 
Testing Service (ETS) for the MFT exam and by 
Center for Computing Education Research 
(CCER) for the ISA exam. Data were analyzed 
using Microsoft Excel 2007 and SPSS 15.0 for 
Windows. A 95% confidence level was used for 
all tests and reported results. 

4. FINDINGS 


Table 1 


F Test for Differences in Two Variances 



F Test Statistic 

1.0798441 



Two-Tail Test 


Lower Critical Value 

0.5214404 

Upper Critical Value 

1.776174 

p-Value 

0.7709159 


To test whether the two groups of students (the 
thirty-one who took only the ISA Exam versus 
the seventy-two who took both the ISA Exam 
and the MFT) are statistically equal, an F-Test 
was run to determine whether the variances in 
the scores are statistically equal. Table 1 
illustrates that the variances are statistically 
equal, so the researchers then ran a two-sample 
t-test that assumes statistically equal variances. 
Table 2 illustrates that mean ISA exams of the 
two groups are statistically equal. Findings of 
Tables 1 and 2 combined indicate that there is 
no anomaly in the two groups, so that the ISA 
Exam scores can be considered to be valid for 
those who took both the ISA Exam and the MFT, 
supporting the null hypothesis for Hi. This 
enables the researchers to continue with further 
comparison of the ISA Exam percentile scores to 
the MFT percentile scores of that group of 
seventy-two CIS students. 


Table 2 


t-Test: Two-Sample Assuming Equal 

Variances 


ISA 

ISA & MFT 

Mean 

60 

58.25 

Variance 

733.13333 

678.725352 

Observations 

31 

72 

Pooled Variance 

694.88614 


Hypothesized 
Mean Difference 

0 


Df 

101 


t Stat 

0.3090365 


P(T<=t) one-tail 

0.3789652 


t Critical one-tail 

1.6600806 


P(T<=t) two-tail 

0.7579304 


t Critical two-tail 

1.9837309 



Table 3 


F Test for Differences in Two Variances 



F Test Statistic 

1.057621 



Two-Tail Test 


Lower Critical Value 

0.625649 

Upper Critical Value 

1.59834 

p-Value 

0.81407 


Moving on to a comparison of the ISA Exam 
percentile scores and the MFT percentile scores 
of the seventy-two students who completed both 
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exams, the researchers first conducted an F-test 0.8235 confirms this, and a coefficient of 

to determine if the variances are statistically determination of R 2 = 0.6782 tells one that 

equal. The results indicate that they are 67.82% of the ISA performance can be 

statistically equal as shown in Table 3. described by the MFT performance. 


Equality of variances leads to the two-sample t- 
test that assumes equal variances for the null 
hypothesis that ISA Exam percentile scores and 
MFT percentile scores are statistically equal for 
the same set of students. The results indicate 
that the percentile scores are indeed statistically 
equal. The results of the t-test are given in 
Table 4. 


Table 4 


t-Test: Two-Sample Assuming Eq 

ual Variances 





ISA 

MFT 

Mean 

58.25 

61.84722222 

Variance 

678.725352 

717.7650626 

Observations 

72 

72 

Pooled Variance 

698.245207 


Hypothesized 
Mean Difference 

0 


Df 

142 


t Stat 

-0.81679776 


P(T<=t) one-tail 

0.20770612 


t Critical one-tail 

1.65565517 


P(T<=t) two-tail 

0.41541223 


t Critical two-tail 

1.97681096 



These results indicate that the ISA exam can be 
utilized as an assessment instrument, as the 
student group who took both exams placed in 
the same percentile for both exams, and has 
been verified over time. This indicates that the 
students' retained knowledge is being measured 
in a parallel manner between disciplines. This is 
not surprising, as both tests are nationally 
normed, and the ISA Exam is built around the 
IS2002 curriculum model, which standardizes 
the coursework and the knowledge base of the 
curriculum. 

As a further exploration and verification of this 
relationship, a scatter plot of the ISA scores 
versus the MFT scores is given in Figure 1 in the 
Appendix. This figure illustrates a positive trend 
in exam performance by students. The slope 
indicates that for every percentage point 
increase in the MFT Exam score, a 0.81 
percentage point increase in the ISA Exam score 
is attained. A correlation coefficient of r = 


Other factors that could drive this relationship 
could be: 

• students' graduating GPA 

• CIS coursework GPA 

• the students' employment status 

• the students' marital/family status 

• students' test-taking ability 

• course repeat for improved GPA 

• multiple majors being pursued 

These variables could increase the R 2 value in a 
multiple linear regression model, and contribute 
to the explanatory nature of the model. 

The only one of the above data to which the 
researches had access was overall GPA. A graph 
of the GPA versus ISA percentile is given as 
Figure 2 in the Appendix. The results are to be 
expected in that a higher GPA indicated a higher 
ISA percentile. Reading the regression statistics 
one can conclude that for every point increase in 
the students' GPA, a 35 percentile point increase 
in the ISA Exam score was accomplished. 

Addition of the graduating GPA to the regression 
model yields: 

y = -32.52 + 0.68 Tj + 13.75 x 2 
where 

y = ISA percentile 
x x = MFT percentile 
x 2 = Graduating GPA 
and R 2 = 0.72 

This multiple regression model now determines 
72% of the variation in the ISA exam percentile 
as a function of the MFT percentile and the 
students' graduating GPA. The remaining 28% 
could be due to the factors listed above, or other 
factors yet to be determined. The slight addition 
to the R 2 value is most likely a result of GPA 
measuring knowledge as it is acquired, while the 
MFT and ISA exams measure knowledge as it is 
retained. 

5. CONCLUSIONS, LIMITATIONS, AND 
RECOMMENDATIONS 

This study demonstrates that the ISA Exam 
indicates students' knowledge retention in the 
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same sense that the MFT does. Therefore, in 
addition to being a means of assessing student 
performance, the ISA Exam can be used as a 
program assessment tool in the same sense the 
MFT is used. 

A limitation of this study lies in the number of 
students involved. While the sample size is 
sufficient to justify these statistical tests and 
conclusions, it is not large enough to draw 
conclusions about all programs. 

Another potential limiting factor is the nature of 
the college in which this study was conducted 
and of the students at that institution. While, no 
anomaly in the college or students is perceived, 
yet the students annually score higher as a 
group than the average score for all who have 
taken the ISA Exam each year. 

A third limiting factor relates to the use of MFT 
overall scores. This study might have yielded 
different results if data were used for an 
information systems subset of questions of the 
MFA. Flowever, such data were not available at 
the subject institution. 

If similar data exist at other institutions, this 
research can be easily replicated. If similar 
results are produced by those replications, the 
limitations of this particular study are overcome. 
The authors encourage the Center for 
Computing Education Research (CCER) to 
sponsor such research efforts. 
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7. APPENDIX 


Figure 1 


MFT versus ISA 


y = 0.8068X + 3.4623 
R 2 = 0.6782 



Figure 2 


GPA versus ISA 


y = 35.409X - 59.332 
R 2 = 0.387 



Graduate GPA 
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